Massively Parallel Data Analysis with PACTs on Nephele

Large-scale data analysis applications require processing and analyzing of Terabytes or even Petabytes of data, particularly in the areas of web analysis or scientific data management. This trend has been discussed as “web-scale data management” in a panel at VLDB 2009. Formerly, parallel data processing was the domain of parallel database systems. Today, novel requirements like scaling out to thousands of machines, improved fault-tolerance, and schema free processing have made a case for new approaches. Among these approaches, the map/reduce programming model [4] and its open-source implementation Hadoop [1] have gained the most attention. Developed for simple logfile analysis, map/reduce systems execute sequential user code in a parallel and fault-tolerant manner, once it has been written to fit the second-order functions map and reduce. However, with the success of map/reduce, many projects have started to push more complex (e.g., SQL-like) operations into the programming model, thereby violating some of its initial design goals (i.e., separation of parallelization and user code) and paying significant performance penalties. To eliminate these shortcomings we have developed the Nephele/PACTs [3, 7] system, a parallel data processor centered around a programming model of so-called Parallelization Contracts (PACTs) and the scalable parallel execution engine Nephele. Our system pursues the same design goals as map/reduce and is highlighted by three properties: